Image capturing apparatus, image processing apparatus, and control methods thereof

ABSTRACT

An image capturing apparatus provided with an image capturing unit that captures a first image, a scene discrimination unit that discriminates a scene of an object a candidate generation unit that generates a shooting setting candidate, based on a result of the discrimination, an image generation unit that generates, based on the shooting setting candidate, an image reflecting an effect obtained by the shooting setting candidate, and a display device. The candidate generation unit performs at least one of a first operation for generating a plurality of shooting setting candidates in which a predetermined parameter is separated by greater than a threshold value and a second operation for generating at least one shooting setting candidate based on the result of the discrimination, and the display device preferentially displays the shooting setting candidates generated by the first operation.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an apparatus that analyzes a shootingscene and presents a user with a plurality of shooting settingsaccording to a result of the analysis.

Description of the Related Art

With a conventional image capturing apparatus, the user obtains adesired image quality by shooting after setting exposure values such asshutter speed and aperture and image quality control values such aswhite balance (hereinafter, WB) control and color parameters to desiredvalues.

Also, recent digital cameras have been proposed that include functionsfor users inexperienced in handling a camera, such as an auto scenefunction that analyzes a scene and then presents an optimal shootingmode or automatically sets color parameters and an exposure correctionvalue, according to the analysis result. Such a camera is described inJapanese Patent No. 3948652, for example.

However, with cameras prior to auto scene functions such as the abovebeing proposed, there was a problem in that, in order to shootpreferable images, specialist knowledge about the shooting functions ofthe camera was essential, making it difficult for a general user toobtain the images he or she wants.

Also, with a camera equipped with an auto scene function, there was aproblem in that since the camera sets an optimal state that depends onthe scene, it is difficult to comprehend what camera settings wereconfigured to obtain the final image, making it difficult for the userto learn the functions of the camera. Such problems occurred not only atthe time of shooting but also in the case of trying to edit images thathave been shot.

SUMMARY OF THE INVENTION

The present invention was made in view of the abovementioned problems,and provides an apparatus that is able to intelligibly display to a userthe relationship between appropriate parameter settings for a scene thatis going to be taken or an image that is going to be edited and an imagethat will be obtained with those settings.

According to a first aspect of the present invention, there is providedan image capturing apparatus comprising: at least one non-transitorymemory device; at least one processor; a camera device that includes ataking lens and an image sensor and captures a first image, based on afirst shooting setting; a scene discrimination unit that discriminates ascene of an object, based on the first shooting setting and a featureamount of the first image; a candidate generation unit that generates ashooting setting candidate, based on a result of the discrimination bythe scene discrimination unit; an image generation unit that generates,based on the shooting setting candidate generated by the candidategeneration unit, an image reflecting an effect obtained by the shootingsetting candidate; and a display control unit that causes a displaydevice to display the image generated by the image generation unit,wherein the candidate generation unit performs at least one of a firstoperation for generating a plurality of shooting setting candidates inwhich a predetermined parameter is separated by greater than or equal toa threshold value out of the shooting settings based on a result of thediscrimination by the scene discrimination unit and a second operationfor generating at least one shooting setting candidate based on theresult of the discrimination by the scene discrimination unit, and thedisplay control unit causes the display device to preferentially displaythe shooting setting candidates generated by the first operation, andwherein the scene discrimination unit, the candidate generation unit,the image generation unit and the display control unit are implementedby the at least one processor executing at least one program recorded onthe at least one non-transitory memory device.

According to a second aspect of the present invention, there is providedan image processing apparatus comprising: at least one non-transitorymemory device; at least one processor; a scene discrimination unit thatdiscriminates a scene of an object, based on a feature amount of a firstimage; a candidate generation unit that generates a setting candidaterelating to at least one of a shooting parameter and an image processingparameter, based on a result of the discrimination by the scenediscrimination unit; an image generation unit that generates, based onthe setting candidate generated by the candidate generation unit, animage reflecting an effect obtained by the setting candidate; and adisplay control unit that causes a display device to display the imagegenerated by the image generation unit, wherein the candidate generationunit performs at least one of a first operation for generating aplurality of setting candidates in which a predetermined parameter isseparated by greater than or equal to a threshold value out of thesettings based on a result of the discrimination by the scenediscrimination unit and a second operation for generating at least onesetting candidate based on the result of the discrimination by the scenediscrimination unit, and the display control unit causes the displaydevice to preferentially display the setting candidates generated by thefirst operation, and wherein the scene discrimination unit, thecandidate generation unit, the image generation unit and the displaycontrol unit are implemented by the at least one processor executing atleast one program recorded on the at least one non-transitory.

According to a third aspect of the present invention, there is provideda method of controlling an image capturing apparatus that includes animage capturing unit that captures a first image based on a firstshooting setting, the method comprising: discriminating a scene of anobject, based on the first shooting setting and a feature amount of thefirst image; generating a shooting setting candidate, based on a resultof the discrimination in the scene discrimination; generating, based onthe shooting setting candidate generated in the candidate generation, animage reflecting an effect obtained by the shooting setting candidate;and displaying the image generated in the image generation, wherein, inthe candidate generation, at least one of a first operation forgenerating a plurality of shooting setting candidates in which apredetermined parameter is separated by greater than or equal to athreshold value out of the shooting settings based on a result of thediscrimination in the scene discrimination and a second operation forgenerating at least one shooting setting candidate based on the resultof the discrimination in the scene discrimination is performed, and inthe display, the shooting setting candidates generated by the firstoperation are preferentially displayed.

According to a fourth aspect of the present invention, there is provideda control method for an image processing apparatus, the methodcomprising: discriminating a scene of an object, based on a featureamount of a first image; generating a setting candidate relating to atleast one of a shooting parameter and an image processing parameter,based on a result of the discrimination in the scene discrimination;generating, based on the setting candidate generated in the candidategeneration, an image reflecting an effect obtained by the settingcandidate; and displaying the image generated in the image generation,wherein, in the candidate generation, at least one of a first operationfor generating a plurality of setting candidates in which apredetermined parameter is separated by greater than or equal to athreshold value out of the settings based on a result of thediscrimination in the scene discrimination and a second operation forgenerating at least one setting candidate based on the result of thediscrimination in the scene discrimination is performed, and in thedisplay, the setting candidates generated by the first operation ispreferentially displayed.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the configuration of an imagecapturing apparatus according to one embodiment of the presentinvention.

FIG. 2A is a block diagram showing signal processing in one embodiment.

FIG. 2B is a block diagram showing signal processing in one embodiment.

FIG. 3 is a diagram showing exemplary settings of a camera shootingfunction.

FIG. 4 is a diagram showing an example of gradation correction settings.

FIG. 5 is a flowchart showing the flow of operations from startup of theimage capturing apparatus until shooting.

FIG. 6 is a flowchart showing operations of shooting parameterdetermination processing.

FIGS. 7A to 7D are conceptual diagrams showing a shooting procedure.

FIG. 8 is a block diagram showing processing for generating a shootingsettings selection screen.

FIG. 9 is a block diagram showing image correction processing.

FIGS. 10A to 10C are diagrams showing exemplary image correctionsettings.

FIGS. 11A and 11B are block diagrams showing defocus processing.

FIG. 12 is a diagram showing an example of performing scenediscrimination.

FIG. 13 is a diagram showing an image divided into blocks.

FIG. 14 is a diagram showing detection settings for respective blockcolors.

FIG. 15 is a diagram showing conditions for brightness determination.

FIG. 16A is a flowchart showing shooting setting proposition processing.

FIG. 16B is a flowchart showing shooting setting proposition processing.

FIG. 16C is a flowchart showing shooting setting proposition processing.

FIG. 16D is a flowchart showing shooting setting proposition processing.

FIG. 16E is a flowchart showing shooting setting proposition processing.

FIGS. 17A and 17B are diagrams showing detection/shooting settings forcolor processing effect.

FIG. 18 is a diagram showing an example of shooting propositioncontents.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, one embodiment of the present invention will be describedin detail with reference to the accompanying drawings.

FIG. 1 is a block diagram showing the configuration of an imagecapturing apparatus according to one embodiment of the presentinvention. In FIG. 1, the image capturing apparatus performs a shootingoperation when a shooting instruction is input by a user via a userinterface (hereinafter, UI) 112. Specifically, a system controller 107adjusts the focal position of a taking lens 102 and controls an imagesensor 101 consisting of a diaphragm, a mechanical shutter, a CMOSsensor and the like to perform shooting. When shooting is performed, ananalog image signal is output from the image sensor 101, undergoes gainadjustment by an AFE circuit (analog front end) 150, and is convertedinto a digital signal. The digital image signal output from the AFEcircuit 150 is stored in a buffer memory 103. Thereinafter, signalprocessing parameters such as a white balance correction coefficient anda gamma parameter are set in a signal processing circuit 140, under thecontrol of the system controller 107. The signal processing circuit 140performs signal processing for image generation on the image signal readout from the buffer memory 103, and YUV image data is generated.

The generated YUV image data is, in the case where image recording isperformed, sent to a compression/decompression circuit 104 to undergocompression processing into a JPEG file, and is recorded to a recordingmedium 106 by a recording apparatus 105. In the case of resizing, sizeconversion processing is performed by a resizing circuit 131. Also, inthe case of displaying an image, the YUV image data stored in the buffermemory 103 is converted into a signal suitable for display by a displaycontrol circuit 108. The signal output from the display control circuit108 is converted into an analog signal by a D/A converter 109, and theresultant image is displayed on a monitor apparatus 110.

Also, in the case of performing camera control using the UI 112 such asconfiguring camera shooting settings, the system controller 107generates a user interface screen. The generated user interface screenis stored in the buffer memory 103, and is displayed on the monitorapparatus 110 via the display control circuit 108. The display controlcircuit 108 is also capable of redundantly displaying YUV image data anda user interface screen. In that case, redundant processing of an imageis performed by an image compositing circuit 133.

Also, in the case of detecting a face from a shot image, a facedetection circuit 120 detects a face from the YUV image data stored inthe buffer memory 103, and outputs the coordinates of the face in theimage. Also, with respect to the region in which the face is detected,image information such as the brightness of the face region can beacquired, by acquiring a histogram of luminance values using a histogramcircuit 130. Also, in the case of dividing the screen into a pluralityof regions and acquiring a histogram for each region, the YUV image datais read out from the buffer memory 103, and the on-screen colordistribution is acquired by an on-screen color distribution acquisitioncircuit 132. In that case, the screen can be divided into a plurality ofregions, an integrated value of YUV can be acquired for each dividedregion, and image data such as saturation, hue and luminance can beacquired from that data. The on-screen histogram is acquirable byanalyzing YUV image data with the histogram circuit 130. In the case ofcorrecting the signal processed YUV image data, an image correctioncircuit 160 performs correction processing.

Next, FIG. 2A is a block diagram showing the flow of image processingfor generating YUV image data from RAW data obtained with the imagesensor 101. When image data is acquired, a WB (white balance) detectionsetting circuit 172 within the signal processing circuit 140 calculatesa WB detection setting value using a WB detection parameter 171,according to the WB mode of the shot image and shooting conditions suchas object luminance (Bv value) and shooting sensitivity. The WBdetection setting value is set in a WB detection circuit 170. In the WBdetection circuit 170, extraction of a white region is performed on theacquired RAW data, based on the set WB detection value. A WB computationcircuit 173 calculates a WB coefficient using the detection result fromthe WB detection circuit 170. WB processing is performed on RAW data bya WB processing circuit 174, using the calculated WB coefficient. On theother hand, color temperature that depends on the detected WBcoefficient can be obtained, by calculating the relationship between WBcoefficient and color temperature in advance. An image processingparameter calculation circuit 180 calculates an image processingparameter according to detected color temperature, color conversionparameter, shooting ISO speed, aperture value and the like. Also, in asignal processing circuit 140A, various processing such as noisereduction, edge enhancement, color reproduction conversion, andgradation conversion is performed, and YUV image data is generated.

FIG. 2B is a block diagram showing the flow of processing in the signalprocessing circuit 140A. RAW data output from the image sensor 101 andconverted into a digital signal is subject to white gain adjustment bythe WB processing circuit 174 and, in a color signal creation circuit141, undergoes processing such as color conversion matrix processing andgradation conversion processing for each color temperature and a colordifference signal is created. The color difference signal obtainedthrough conversion performed by the color signal creation circuit 141 issubject to RGB conversion, and undergoes color gradation conversion by acolor gradation conversion circuit 145. The RGB signal that hasundergone color gradation conversion is again converted into a colordifference signal, and is subject to saturation adjustment by asaturation enhancement circuit 146.

Meanwhile, the signal from the WB processing circuit 174 undergoesinterpolation processing, edge enhancement processing and noisereduction processing that depend on ISO speed, shooting parameters,aperture value and the like by a luminance signal creation circuit 142and a luminance signal is created. Thereinafter, the luminance signalundergoes gradation conversion by a luminance gradation conversioncircuit 143 and correction by a luminance correction circuit 144according to the hue of the luminance signal, and is output as aluminance signal.

FIG. 3 is a diagram showing an example of camera shooting settings. Anexample is shown in which nine types of settings, namely, brightness,motion blur, background bokeh, coloration setting, coloration correction(amber-blue), coloration correction (green-magenta), crispness,vividness and finishing correction, for example, are given as settableitems. Shooting control values and signal processing parameters are set,according to these camera settings.

In FIG. 3, brightness correction sets an exposure correction value withrespect to an exposure control value determined by AE (auto exposurecontrol). The control value is changed according to the set correctionvalue, based on a program diagram. In the present embodiment, an exampleis shown in which the correction value can be set over ±9 steps from −3to +3 in ⅓ stop increments.

A background bokeh setting represents setting of the aperture value atthe time of shooting, with depth of field becoming shallower andbackground bokeh becoming greater as the aperture is opened up. Thisfunction changes the ISO speed value and the aperture setting accordingto the shutter speed setting to change background bokeh whilemaintaining brightness. An example is shown in which the shutter speedsetting is settable over ±6 steps from −2 to +2 in ⅓ stop increments,with the number of stops indicating the difference from the shutterspeed at the correct exposure. −6 refers to a setting for shooting witha shutter speed 2 stops slower than normal.

The coloration setting refers to a basic white balance setting. Autorepresents an automatic white balance setting, and involves analyzing animage and configuring an optimal white balance setting for the image.The white balance setting other than auto is a WB setting adjusted forindividual light sources, and is a setting configured by calculating WBsetting values (white balance setting values) in advance according tothe respective light sources.

Coloration correction (G−Mg) and coloration correction (A−B) arefunctions that finely adjust the white balance setting, by adding acorrection value to the WB setting value determined by the colorationsetting. Here, the respective WB processing is represented by thefollowing equations, where R, G1, G2 and B are the sensor output valuesof a gray object that are based on the color filter array of the imagesensor, and WB_R, WB_G1, WB_G2 and WB_B are the WB coefficientscorresponding to the respective colors.G1′=G1×WB_G1R′=R×WB_RB′=B×WB_BG2′=G1×WB_G1Here, white balance will have been set appropriately when R′, G1′, G2′and B′ take the same values.

Also, the WB evaluation values are represented as follows, where Cx andCy are the WB evaluation values with respect to the WB coefficients.Cx=(R−B)/Y×1024Cy={R+B−(G1+G2)}/Y×1024Y=(R+G1+G2+B)/4Here, the values of Cx and Cy are calculable from the WB coefficientsWB_R, WB_G1, WB_G2 and WB_B, and, conversely, the WB coefficients can becalculated from Cx and Cy, by assuming that the average value of the WBcoefficients is 1024. Also, WB correction can be performed using thevalues of Cx and Cy, by correcting the values of Cx and Cy as follows,where ΔCx and ΔCy are the WB correction values with respect to Cx andCy, a is the WB correction setting value (A−B), and β is the WBcorrection setting value (G−Mg).Cx′=Cx+α×ΔCxCy′=Cy+β×ΔCy

In the present embodiment, α and β are described as being settable over±9 steps.

The finishing setting represents a standard image quality setting thatdepends on the scenes of each camera, and, in the present embodiment,five types of settings, namely, Standard, Landscape, Portrait, Neutraland Faithful, are available. With these settings, parameters are setsuch that the settings of the color signal creation circuit 141, thecolor gradation conversion circuit 145, the saturation enhancementcircuit 146, the luminance gradation conversion circuit 143 and theluminance correction circuit 144 are optimized for each setting,according to the respective scenes.

Generally, in Standard, vividness and contrast are set slightly high tocreate a striking picture, and, in Landscape shooting, contrast andvividness are set higher than in Standard. Also, in Portrait, skin coloris set slightly brighter and saturation is set lower than in Standard togive a soft luminance gradation in brighter regions. In Neutral,saturation and contrast are set lower than in Standard, and, inFaithful, parameters are set so that the actual color tone of the objectcan be faithfully realized.

Crispness can be adjusted by changing the gamma characteristics that areset with the luminance gradation conversion circuit 143. FIG. 4 showsthe gradation characteristics that are set by the luminance gradationconversion circuit 143, with the gradation characteristics overcontrasts from −4 to +4 being shown. The feeling of contrast in theimage can be intensified by making the dark regions darker and causinghalation to occur early.

The vividness setting is adjusted with the image processing parametersthat are set with the saturation enhancement circuit 146 of FIG. 2B, andis represented by the following equations.U′=(1+vividness setting value×ΔGain)×UV′=(1+vividness setting value×ΔGain)×VHere, ΔGain is a parameter that controls the enhancement amount ofsaturation that depends on the vividness setting, and a vividnesssetting of 0.6 to 1.4 times the standard saturation setting may beconfigured when the setting value can be changed from −4 to +4 withΔGain set to 0.1, for example.

FIG. 5 is a flowchart showing the flow of operations from startup of theimage capturing apparatus until shooting in the present embodiment. Instep S101, live view display is performed. In step S102, it is detectedwhether power is turned OFF, and if power is in an OFF state, the imagecapturing apparatus is shut down and camera operations are ended. Ifpower is in an ON state, the processing proceeds to step S103. In stepS103, it is determined whether an Analyze button has been pressed. Ifthe Analyze button has been pressed, the processing proceeds to stepS200, and shooting parameter determination processing which will bediscussed later is performed. In the present embodiment, three types ofshooting parameters that are determined at step S200 will be described.In step S105, if it is detected that shooting effect settings have beenselected or changed, the shooting effect settings are changed, and theprocessing proceeds to step S300 and processing for reflecting theshooting parameters is performed.

Here, the shooting parameters include shutter speed, aperture value,exposure correction value, ISO speed, contrast, sharpness, saturation,color tone, white balance setting and filter effect. Changing theshutter speed, aperture value and exposure correction value is performedby changing the control parameters of the taking lens 102 and thecontrol of the image sensor 101. Changing the ISO speed is performed bygain control of the AFE 150. Changing the contrast, sharpness, colortone and filter effect is performed by controlling the image processingparameter calculation circuit 180 to change development settings withrespect to the signal processing circuit 140A. Changing the whitebalance setting is performed by controlling the WB computation circuit173 and changing the operations of the WB processing circuit 174.

In step S107, it is determined whether a switch SW1 that is turned ON byhalf pressing a release button has been turned ON. If there is ashooting standby instruction resulting from the switch SW1 being turnedON, the processing proceeds to step S108 and shooting preparation isperformed. Specifically, light metering by AE (auto exposure control)and focal detection by AF (autofocus) are performed, and actual shootingconditions are determined. Also, in the shooting standby state of stepS109, when a switch SW2 is turned ON by the release button being fullypressed (step S110: YES), the processing proceeds to step S112. In stepS112, actual shooting is performed under the determined shootingconditions, assuming that actual shooting has been instructed. On theother hand, if, at step S110, the switch SW2 has not been turned ON, theprocessing proceeds to step S111 and it is determined whether the switchSW1 has been released. If, in step S111, the switch SW1 has not beenturned OFF, the processing returns to step S109 and the shooting standbystate is continued. Also, if the switch SW1 has been turned OFF and theshooting standby state has been canceled, the processing returns to stepS101 and the live view display state is entered.

FIG. 6 is a flowchart showing operations of shooting parameterdetermination processing in step S200 of FIG. 5. When it is determinedin step S103 of FIG. 5 that the Analyze button has been pressed,shooting for scene analysis is performed in step S201. Shooting settingsfor scene analysis are preferably standard settings not based onshooting settings set by the user beforehand. For example, to detectbacklight or night view, screen histogram analysis or the like istypically performed, in which case the brightness of the screen being atthe correct level is necessary in order to perform scene discriminationcorrectly. Standard settings are similarly desirable even in the case ofanalyzing AF or color reproduction.

In step S202, the AF operation is performed and the main object isbrought into focus. As the AF (autofocus) technique, focus detectionemploying a well-known phase difference detection method or the like canbe used. At step S203, AE (auto exposure control) is performed andstandard brightness is set, and at the same time WB detection and AWB(auto white balance control) processing are performed.

In step S204, an image for scene analysis is shot using the shootingsettings set at steps S202 to S203. In step S205, scene analysis isperformed. Scene analysis involves analyzing items such as thefollowing, for example. First, it is determined through face detectionwhether there are any persons. Distance information to the object iscalculated from the AF result, and macro shooting is discriminated.Night view, evening view, landscape, indoor scene or the like isdiscriminated from the AE result, the tone color and luminance values onthe screen, the AF distance information and the like. Backlight isdiscriminated from object luminance values (Bv values) and the on-screenhistogram. On-screen color distribution and the like are acquired fromthe on-screen histogram result, and the main object is discriminated.

In step S206, three types of shooting parameters are determined,according to the scene analysis result. In step S207, processing forreturning the shooting settings for analysis set in step S201 to theoriginal shooting settings is performed.

FIGS. 7A to 7D are conceptual diagrams showing a shooting settingcandidate being selected on a selection screen and reflected in the liveview image, after the Analyze button has been pressed in step S103 ofFIG. 5, in the present embodiment. The scene in FIGS. 7A to 7Drepresents a backlight state, and is assumed to be a shooting scene inwhich the background is bright and the object is slightly dark.

FIG. 7A shows the Analyze button being displayed at the same time whiledisplaying the live view image. FIG. 7B shows a state in whichcandidates for three types of shooting settings (candidate generation)that depend on the scene and images of the image effects when thosesettings are configured are displayed, as a result of pressing theAnalyze button in FIG. 7A. In FIG. 7B, a setting candidate 1 shows theperson as being darkened as a result of reducing the exposure andsetting the background to the correct exposure. A setting candidate 2shows the person as being correctly shot and the background as beingbrighter, as a result of increasing the exposure. A setting candidate 3shows the exposure of the background as being suppressed to an extentthat halation does not occur, as a result of reducing the exposure onestop, and the person as being slightly brighter, as a result of loweringthe contrast and lifting the dark portion of the dark region.

The image diagrams displayed here are displayed by performing imageprocessing on the image captured when the Analyze button is pressed, soas to approach images that would be obtained with the above shootingsettings. It is also possible to perform this image processing on thelive view image and display the result by installing a high-speedprocessing circuit as hardware.

In FIG. 7B, when the setting candidate 1, for example, is selected bythe user, shooting is performed with the shooting setting of the settingcandidate 1, and the shot image is displayed as a live view image. Thelive view image referred to here is obtained by displaying an image shotafter actually setting the exposure, without correction by the imageprocessing circuit such as aforementioned. Also, as shown in FIG. 7C,the shooting setting is displayed on the monitor apparatus 110, and itis also possible for the user to further change the settings from thisshooting setting. FIG. 7D shows a state in which a shooting instructionhas been given in FIG. 7C and a review image after performing stillimage shooting is displayed.

FIG. 8 is a block diagram showing the flow of operations for generatingthe selection screen for the three types of shooting settings in FIG.7B. The YUV image data obtained with the image acquisition operation forscene analysis in step S204 of FIG. 6 undergoes scene discrimination bya scene analysis circuit 200 that is realized by the system controller107. The scene analysis circuit 200 is realized by the system controller107 performing computations, based on information that is obtained fromshutter and aperture control values output to the face detection circuit120, the histogram circuit 130, the on-screen color distributionacquisition circuit 132 and the taking lens 102 in FIG. 1, and the like.A shooting setting determining circuit 210 that is realized by thesystem controller 107 determines three types of shooting settings,according to scene information determined by the scene analysis circuit200. An image processing setting circuit 220 that is realized by thesystem controller 107 determines the image processing parametersaccording to the determined shooting settings. The determined imageprocessing parameters are image processing parameters that are used inthe processing of FIG. 9 which will be discussed later.

On the other hand, the YUV image data is resized by the resizing circuit131 to a size for displaying the influence of the parameter change,undergoes three types of image processing by the image correctioncircuit 160, and is composited with the original YUV image data by theimage compositing circuit 133. Here, the image conversion resulting fromchanging the image processing parameters can be generated by the signalprocessing circuit 140 from a RAW image. However, as already describedwith FIGS. 7A to 7D, this image conversion can be generated in a pseudomanner, by performing simple image processing on the analyzed image.Since an image to which a plurality of effects have been provided canthus be generated from the same image with simple processing, there areadvantages in that a screen for comparison of shooting settings andselection of shooting settings by the user can be readily displayed.Also, a RAW image does not need to be saved in order to generate thisimage conversion from a YUV image, enabling processing speed and memoryusage to be reduced, and facilitating simplification of system control.

FIG. 9 is a block diagram showing the flow of processing in the imagecorrection circuit 160. A YUV/RGB conversion circuit 161 converts YUVimage data into RGB image data. A tone curve correction circuit 162performs tone curve correction for each of RGB. Tone curve correctioninvolves performing exposure correction processing, contrast correctionprocessing, and processing for changing WB by changing the tone curvefor each of RGB.

A 3×3 matrix circuit 163 is capable of color reproduction, colorationconversion and saturation correction, which involve processing forapplying a 3×3 matrix to RGB signals. A defocus processing circuit 164is capable of background defocus processing for defocusing an imageperipheral part and diorama-like filter processing for controlling thedefocus amount for each vertical line of the image. The defocusprocessing circuit 164 is also additionally capable of toy camera-likeprocessing for lowering the peripheral brightness and cross filter-likeprocessing for applying a cross filter to bright portions. An RGB/YUVconversion circuit 165 converts RGB image data into YUV image data.

FIGS. 10A to 10C are diagrams showing an example of image correctionsettings configured by the image correction circuit 160 shown in FIG. 9.The image correction circuit 160 is an image processing circuit thatperforms pseudo generation of image effects in the case where theshooting settings are changed on YUV image data that has been developednormally. Processing for changing the shooting settings in the presentembodiment includes exposure correction processing, contrast correctionprocessing, white balance change processing, depth-of-color changeprocessing, hue change processing and background defocus processing.

FIG. 10A is a diagram showing the characteristics of a tone curve forexposure correction. The tone curve for exposure correction is generableby the following computational equation.AECorrect(x)=γ(γ⁻¹(x)×2^(exposure correction value))

Here, γ(x) indicates a standard gradation conversion characteristic thatis set by the luminance gradation conversion circuit 143 shown in FIG.2B, where γ⁻¹(x) indicates the inverse transform characteristic of γ(x).

FIG. 10B is a diagram showing the characteristics of a tone curve forcontrast correction.ContCorrect(x)=γi(γ0⁻¹(x))

Contrast setting value i=−4 to +4 Here, γi( ) indicates the gradationconversion characteristic in the case where contrast correction, whichis set by the luminance gradation conversion circuit 143 shown in FIG.2B, has been set, and the suffix i indicates the setting value. Withregard to the value of i, 0 indicates the standard, the + directionindicates a conversion that strengthens contrast, and the − directionindicates a conversion that weakens contrast.

FIG. 10C is a diagram showing the characteristics of a tone curve for WBcorrection. In the present embodiment, the respective tone curves forRGB are calculated using a ratio of the AWB coefficient used at the timeof RAW shooting and the corrected WB coefficient.GainR=WB_R_corrected/WB_R_AWBGainR=WB_G_corrected/WB_G_AWBGainR=WB_B_corrected/WB_B_AWBWBCorrectR(x)=cγ(cγ ⁻¹(x)×GainR)WBCorrectG(x)=cγ(cγ ⁻¹(x)×GainG)WBCorrectB(x)=cγ(cγ ⁻¹(x)×GainB)Here, cγ( ) is a gamma curve for color and indicates the gradationcharacteristics that are used by the color gradation conversion circuit145 shown in FIG. 2B. In the present embodiment, the WB coefficientratio illustrates curves in 0.1 increments, and the gradation conversioncharacteristics of gains of 0.1 or less may be determined using linearinterpolation.

The gradation characteristics shown in FIGS. 10A to 10C are compositedin order of exposure correction, contrast correction and WB correction,and the composited tone curve characteristics are respectively set inthe tone curve correction circuit 162 of FIG. 9.

Next, techniques for saturation correction and hue correction will bedescribed. In the case of JPEG data, the following relation equation isdefined with a color difference signal and an RGB conversion equation.

$\begin{pmatrix}Y \\{R - Y} \\{B - Y}\end{pmatrix} = {M\begin{pmatrix}R \\G \\B\end{pmatrix}}$ $M = \begin{pmatrix}0.3 & 0.59 & 0.11 \\0.7 & {- 0.59} & {- 0.11} \\{- 0.3} & {- 0.59} & 0.89\end{pmatrix}$The RGB signal is derived as follows from the YUV signal, using aninverse matrix.

$\begin{pmatrix}R \\G \\B\end{pmatrix} = {M^{- 1}\begin{pmatrix}Y \\{R - Y} \\{B - Y}\end{pmatrix}}$Color difference signal correction processing, which is correctionprocessing on the YUV color difference signal, can be represented asfollows when the saturation enhancement parameter is given as a.

$\begin{pmatrix}Y \\{R - Y} \\{B - Y}\end{pmatrix} = {K\begin{pmatrix}Y \\{R - Y} \\{B - Y}\end{pmatrix}}$where

$K = \begin{pmatrix}1 & 0 & 0 \\0 & \alpha & 0 \\0 & 0 & \alpha\end{pmatrix}$In other words, the saturation correction parameter can be representedwith the following equation.

$\begin{pmatrix}R^{\prime} \\G^{\prime} \\B^{\prime}\end{pmatrix} = {M^{- 1} \cdot K \cdot {M\begin{pmatrix}R \\G \\B\end{pmatrix}}}$

Hue angle conversion can also be represented as follows when therotation angle is given as θ.

$\begin{pmatrix}Y \\{R - Y} \\{B - Y}\end{pmatrix} = {P\begin{pmatrix}Y \\{R - Y} \\{B - Y}\end{pmatrix}}$where

$P = \begin{pmatrix}1 & 0 & 0 \\0 & {\cos\;\theta} & {\sin\;\theta} \\0 & {{- \sin}\;\theta} & {\cos\;\theta}\end{pmatrix}$such that

$\begin{pmatrix}R^{\prime} \\G^{\prime} \\B^{\prime}\end{pmatrix} = {M^{- 1} \cdot P \cdot {M\begin{pmatrix}R \\G \\B\end{pmatrix}}}$Hue angle conversion and saturation enhancement parameters can becombined and expressed as follows.

$\begin{pmatrix}R^{\prime} \\G^{\prime} \\B^{\prime}\end{pmatrix} = {Q\begin{pmatrix}R \\G \\B\end{pmatrix}}$where Q=M⁻¹·P·K·M.

It thus becomes possible to perform color correction by setting thecombined computation matrix Q in the 3×3 matrix circuit 163. It is ofcourse possible to represent not only saturation enhancement and hueangle conversion but also the conversion equation for color reproductionof various landscapes or a portrait with a 3×3 matrix, and imageconversion processing can be performed by combining this conversionequation with the aforementioned matrix Q.

FIGS. 11A and 11B are block diagrams showing the procedure of processingby the defocus processing circuit 164 shown in FIG. 9. Defocusprocessing is performed by a low pass filter or a Gaussian filter beingapplied to image data by a defocus processing circuit 166. An image thathas not undergone defocus processing is multiplied by a composite maskrepresenting the image composition ratio for each pixel, whereas with animage that has undergone defocus processing, the composition ratio ofthe composite mask is subtracted from 1, and both images are composited.FIG. 11B shows an example of a composite mask for creating the peripheryof the image into a defocused image. These composite masks may bedetermined from the position of the main object by detecting the mainobject. Also, although a binarized composite mask is shown in thisexample, a configuration may be adopted in which composition processingis smoothly performed by gradually changing the composition ratio. Inthis case, generation is possible by applying a defocus processingfilter to the composite mask.

FIG. 12 is a diagram showing an example of scene discrimination in thepresent embodiment. Hereinafter, the method of scene analysis in stepS205 of FIG. 6 will be described. In the present embodiment, a facedetection result, a motion vector, object distance, on-screen distanceinformation, object luminance, in-plane luminance distribution, in-planecolor distribution and intra-image contrast, for example, are used, inorder to perform scene discrimination. Hereinafter, a method ofcalculating information that is used in each of the aforementioned scenediscriminations will be described.

Face detection methods include methods using learning that are generallyrepresented by neural networks, and methods of detecting a region havinga characteristic physical shape such as the eyes or nose from an imageregion using template matching. Many other techniques such as detectingimage feature amounts like skin color and eye shape and performingstatistical analysis have also been proposed, and generally a pluralityof these methods are used in combination. Furthermore, methods such aface detection method utilizing wavelet transform and image featureamounts and a method that combines template matching and the like havebeen proposed. In any case, detection of a face from an image isimplementable with many well-known methods, and thus a detaileddescription is omitted in the present embodiment.

Main object motion detection is performed by, for example, judgingwhether the position of the detected face moves a predetermined distanceor more between a plurality of frames. Apart from face detection, motionmay be determined using the amount of movement between frames of thepoint of focus determined by autofocus. Also, a technique that involvesperforming template matching in time series with regard to an objectdesignated by the user in an image on the UI 112 or the monitorapparatus 110 and determining motion from the amount of movement betweenframes may be used. Many propositions have also been made with regard tothis determination of the motion of the main object, and a detaileddescription is omitted.

The in-plane color distribution data is obtained by the on-screen colordistribution acquisition circuit 132 dividing the YUV image data that isoutput from the signal processing circuit 140 in FIG. 2 into a pluralityof blocks, and calculating the average value of YUV for each block. FIG.13 is a diagram showing YUV image data divided into a plurality ofblocks in a plane.

Saturation S and hue H are calculated using the following equation fromthe YUV values for each in-plane block.

Luminance Y Hue H = tan⁻¹ (V/U) Saturation S = √(U² + V²)The data of YHS for each block is in-plane color distribution data.

FIG. 14 is a diagram representing detection threshold values ofluminance, hue and saturation of each block in block data with upperlimits and lower limits of scene determination conditions. The number ofblocks that are included in each threshold value is calculated, and theratio thereof to all of the blocks on the screen is calculated. Thescene discrimination level is calculated by comparing the calculatedratio with the numerical values of a screen ratio L and a screen ratioH. In the case where the calculated ratio is less than the screen ratioL, the color scene discrimination level is set to 0, in the case wherethe calculated ratio exceeds the screen ratio H, the color scenediscrimination level is set to 100, and in the case where the calculatedratio is between the screen ratio L and the screen ratio H, the colorscene discrimination level is calculated using linear interpolation.Object luminance (brightness value; hereinafter, Bv value) is calculatedfrom brightness at the time that light metering is performed in stepS203 of FIG. 6. Since there are many well-known examples regarding AEtechniques, a detailed description is omitted. In the presentembodiment, description will be given based on a general AE technique.

A block integrated value is acquired from the image acquired at the timeof AE light metering, and a block luminance value Y[i] is calculated.Y[i]=0.3*R[i]+0.3*G1[i]+0.3*G2[i]+B[i]

-   -   (i: block position)

The block luminance value is multiplied by a weight of the main objectfor each block or a center-weighted weight AE_Weight, and a luminancevalue Y_Ave is calculated.Y_Ave=Σ(AE_Weight[i]×Y[i])

An exposure level difference DeltaBv with a light metering value iscalculated, such that the calculated luminance value achieves an AEtarget value (AE_Target).DeltaBv=log₂(Y_Ave/AE_Target)

The object luminance is calculated by adding the calculated exposurelevel difference to the exposure value at the time of light metering.Bv=Av+Tv−Sv+DeltaBv

An Apex value that is used in the above exposure calculation iscalculable using definitions.Av:Av=log₂ F ² F: FNoTv:Tv=log₂(1/T) T: Shutter speedSv:Sv=log₂(S×0.32) S: ISO speed value

The following equation holds at the time of correct exposure.Av+Tv=Bv+Sv

The aperture value, shutter speed and sensitivity are calculated fromthe calculated object luminance value, based on a program diagram set inadvance.

Light metering methods in the case of calculating correct exposureinclude evaluation metering, center-weighted metering, average metering,and spot metering. With methods such as evaluation light metering,exposure conditions are determined, in the case where it is detectedthat the main object is a face, such that the brightness of the face,for example, will be correct.

Also, a block Bv value (block luminance value) is also calculable, bycalculating the luminance difference from the correct level for eachblock, with respect to the Bv value detected at the time of lightmetering. The block luminance value is represented by the followingequation, where BlockY is the block integrated value or the calculatedluminance value and AE_Target is the AE target value.BlockBv[i]=Bv+log₂(BlockY[i]/AE_Target)

-   -   (i indicates the block position)

The object distance is calculated from the result of AF (autofocus).Autofocus methods include a contrast detection method and a phasedifference detection method. Phase difference detection methods alsoinclude a method that is able to calculate the object distancethroughout the entire screen, using an image sensor in which pixels thatdetect phase difference are disposed on an image capturing surface.

Any of the contrast detection methods and the phase difference detectionmethods enable the distance to the object to be calculated using thefocal length of the taking lens and the aperture value, by dividing theimage region into a plurality of blocks and calculating a focus positionfor each region. The object distance can be calculated for all the blockregions on the screen, and set as on-screen distance information. In thecase where the distance to the main object is calculated using any ofthese methods, methods for discriminating the main object include thefollowing. For example, a face region on the screen is determined to bethe main object. Alternatively, a close-range object near the screencenter, an object of a different color to the periphery, an object setby the user using the screen of the monitor apparatus 110 or the like isdetermined to be the main object. The main object can be discriminatedusing these methods, and the distance thereto can be detected.

An intra-image contrast value is calculated from the histogram resultthat is calculated by the histogram circuit 130. The difference inluminance value between the top 10% and the bottom 10% of luminancevalues of the histogram or a standard deviation value that is calculatedfrom the histogram can be set as a contrast value.

Techniques for determining the camera state include a method using anangular velocity sensor. It can be determined, from the output of theangular velocity sensor, that the camera is mounted on a tripod in thecase where the camera is stationary, is in a panning state in the casewhere the camera is moving in a constant direction, and is in ahand-held shooting state in the case where the camera is movingdiscontinuously. Many of the latest digital cameras now have acamera-shake correction mechanism that uses an angular velocity sensorsuch as a gyroscope sensor, and such sensors can be appropriated indetermining the camera state.

Hereinafter, the scene discrimination method shown in FIG. 12 will bedescribed.

Portrait Determination

In portrait determination, it is determined, from the detection resultof the face detection circuit 120 in FIG. 1, that the scene is aportrait scene, in the case where the face is greater than or equal to agiven size in proximity to the center of the screen and where the numberof faces is two or less. More specifically, it is determined to be aportrait scene, in the case where the value of the portrait level iscalculated based on the position of the face, the size of the face, theface detection reliability, the number of persons and the like and isgreater than or equal to a predetermined value.Portrait level=face position weight×face size weight×facereliability×number-of-persons weightFace position weight: center→100%, 80% or more of image height→0%Face size weight: 20% or more of angle of view→100%, 5% or less of angleof view→0%Degree of face reliability: 0%-100%Number-of-people weight: 1→100%, 2→50%, 3→30%, 4 or more→0%

Here, in the case of determining the face position weight, importance iscalculated, with respect to each of a plurality of detected faces, forexample, from the center position and size of the face, and the faceposition weights are determined, with the face having the highest faceimportance as the main face and the other faces as secondary faces.

The main object motion detection can be performed by calculating theaforementioned object motion amount. Also, acquisition of the image dataof the plurality of frames that are used by the technique using imagecorrelation may be performed by acquiring a plurality of frames afterthe Analyze button is pressed in step S103 of FIG. 5. Alternatively, animage may be acquired beforehand during live image display, and scenediscrimination may be performed depending on the determination performedat the time that the Analyze button is pressed.

Object Motion Determination

In the case where there is motion greater than or equal to apredetermined value, according to the motion amount of the main object,that object may be determined as the motion object. Also, it is possibleto calculate the optimal shutter speed without producing object blur,from the relationship between the motion amount and the data acquisitiontime period between frames.

Landscape Determination

Landscape determination is performed using the object luminance obtainedby performing light metering in step S203 of FIG. 6, and the data of theon-screen color distribution acquisition circuit 132. As thedetermination conditions, the object luminance being greater than orequal to a predetermined value and the number of green blocks and blueblocks obtained by in-plane tone color determination, for example, areused.

Macro Determination

Macro determination is performed using the distance of the main objectthat is calculated from AF. In the case where the object distance isless than a predetermined ratio with respect to the focal length of thetaking lens 102, such as the object distance being less than 20 timesthe focal length, for example, it is determined to be macro shooting. Inthe case of a distance greater than or equal thereto, the macrodiscrimination level is calculated according to the distanceinformation, with the macro determination level set to 0 in the casewhere the object distance is up to 40 times the focal length, forexample. The multiplying factor of the focal length may be changedaccording to the aperture value of the taking lens 102.

Dynamic Range Determination

Dynamic range is calculated with respect to the shooting scene. In thepresent embodiment, the number of pixels greater than or equal to apredetermined luminance is counted from contrast data, and it is judgedthat the dynamic range on the bright side is insufficient in the casewhere the counted number is greater than or equal to a predeterminednumber. Conversely, it is judged that the dynamic range on the dark sideis insufficient, in the case where the number of pixels less than orequal to a predetermined luminance value is counted and the countednumber is greater than or equal to a predetermined number. Also, thedeficient amount of the dynamic range on the over side and the underside may be estimated, according to the counted number of pixels.

Backlight Determination

In backlight determination, the difference between the average value ofblock Bv values for a center portion in the image or the main objectdetection portion and the average value of block Bv values for an imageperipheral portion is calculated, and it is judged to be backlit in thecase where the background is brighter by greater than or equal to apredetermined amount. Also, in the case of determining the backlightlevel, the determination can be performed with, for example, thebacklight level set to 0% in the case where the above difference is lessthan 2 stops and the backlight level set to 100% in the case where theabove difference is greater than or equal to 4 stops.

Brightness (Outdoors/Indoors) Determination

Brightness determination is performed using object luminance. In thecase where the object luminance is greater than or equal to apredetermined value, such as Bv≥5, for example, it can be determined tobe outdoors, and in the case where Bv<5, it can be determined to beindoors. Apart from object luminance, in the case where there are manygreen and blue objects when the in-plane color distribution is viewed,it may be determined to increase the level of the outdoor determination.For example, the outdoors level is set to 100% at or above the Bv upperlimit and is set to 0% at or below the Bv lower limit according to thenumber of blue and green blocks, as shown in FIG. 15, and in the case ofan intermediate Bv, linear interpolation is performed and the outdoorslevel is calculated.

Blue Sky Determination

Blue sky determination is performed using Bv values and in-plane colorinformation. For example, it is determined to be blue sky in the case ofsatisfying either of the following conditions:

Condition 1: Bv≥7, blue block≥40%

Condition 2: Bv≥5, blue block≥20%, white block 30%

Also, the blue sky level is calculated by the ratio of the number ofblocks that are detected with condition 1 and condition 2.

Night View Determination

In night view determination, it is determined to be a night view in thecase where the Bv value is less than or equal to 0, and the contrastvalue of the histogram is greater than or equal to a predeterminedvalue. The night view level is calculated such that the value increasesas the Bv value decreases and the contrast value increases.

Evening View Determination

Evening view determination is performed using Bv values and in-planecolor information. It is determined to be an evening view, in the caseof satisfying either of the following conditions:

Condition 1: 2≤Bv≤5, orange block≥40%, blue block≥20%, contrast value≥30

Condition 2: 5≤Bv≤10, orange block≥20%, blue block≥40%, contrastvalue≥30

It is also possible to calculate the evening view level according to thenumber of orange blocks.

Spotlight Determination

In spotlight determination, the difference between the average value ofblock Bv values of an image center portion or the main object detectionportion and the average value of block Bv values of an image peripheralportion is calculated, and in the case where the peripheral luminancevalue is lower by 4 stops or more, for example, it is determined to be aspotlight. Also, the spotlight level may be calculated according to thesize of this difference.

Background Distance

Calculation of the background distance is performed using the on-screendistance map which has already been described. A histogram of objectdistances for each block may be generated, and the distance of theregion having the highest appearance frequency may be set as thebackground object distance, out of the regions that are separated by ata depth of 5 or more by depth conversion, for example, with respect tothe main object distance. Alternatively, a region that is furthest away,out of blocks of a predetermined number or more, may be set as thebackground distance. Setting blocks of a region within a predetermineddistance from the background distance as the background region enablesthe main object region and the background region to be set and theobject distance to be calculated.

Color Scene Determination

In color scene determination, the color scene discrimination level ofyellow, blue, red and green are calculated. The in-plane block numberthat satisfies a predetermined condition from the hue and saturationcalculated from the aforementioned block integrated values is extracted,and the color scene discrimination level is calculated based on thisnumber.

Next, the method of determining the shooting parameters in step S206 ofFIG. 6 will be described. The description in this embodiment is premisedon three types of shooting parameters being presented according to thescene analysis result. FIG. 16A is a flowchart showing the flow ofoperations from scene discrimination to determination of shootingproposition contents.

In FIG. 16A, processing for extracting pair proposition contents isperformed in step S301. Pair proposition refers to presenting two typesof shooting settings that are contrasted. By presenting two types ofshooting settings that are contrasted, the difference in shootingsettings can be emphasized, and the user can be made aware of thedifference in effect between the shooting settings.

FIG. 16B is a flowchart showing the flow of operations for selecting apair proposition setting. In step S401, portrait scene determination isperformed. With a portrait scene, two types of shooting settings inwhich the parameters are separated by greater than or equal to athreshold value, namely, a shooting setting for performing shooting thatdefocuses the background by reducing the depth of field and a shootingsetting for crisply capturing the background by increasing the depth offield, can be proposed.

The depth difference effect fitness is calculated, according to theportrait level of the face detection result and the depth differencebetween the main object and the background object that is obtained fromthe difference between the main object distance and the backgroundobject distance. It is judged to perform portrait proposition in thecase where both the face detection result and the depth differencefitness are greater than or equal to a predetermined value. In the casewhere it is judged at step S401 to perform portrait proposition, asetting in which the aperture value is set to F2.8 and a setting inwhich the aperture value is set to F11.0 are both proposed as a pair instep S402.

In step S403, it is determined whether it is possible to propose macroshooting. In macro shooting, a shooting method that defocuses thebackground and a shooting method that also does not defocus thebackground can be proposed. In macro proposition determination, themacro determination result and the depth difference between the mainobject distance and background distance are calculated, and the depthdifference fitness is calculated. It is judged that macro proposition ispossible in the case where the macro determination result and the depthdifference fitness are greater than or equal to a predetermined value.In the case where it is judged at step S403 that macro proposition ispossible, a setting in which the aperture value is set to F2.8 and asetting in which the aperture value is set to F11.0 are both proposed asa pair in step S404.

In step S405, backlight shooting fitting scene determination isperformed. With a backlight scene, a method of shooting with thebrightness of the main object set to a predetermined brightness and amethod of shooting with the main object set to be in silhouette stylecan be proposed. Backlight proposition is performed based on backlightdetermination which has already been described. In the case where it isjudged at step S405 that backlight proposition is possible, in stepS406, DeltaBv is derived with respect to each of the blocks of the mainobject region and the background region, and an exposure setting valueis determined. An exposure correction setting value is calculated fromthe calculated exposure setting value.

Next, single proposition extraction is performed in step S302 of FIG.16A. Single proposition is processing for determining fitness andshooting settings suitable for the scene, based on analyzed sceneinformation.

FIG. 16C is a flowchart showing the flow of the single proposition instep S302 of FIG. 16A. In FIG. 16C, in step S501, the scenediscrimination level of the portrait scene is determined, and, in thecase where the discrimination level is greater than or equal to apredetermined threshold value, shooting settings for portrait shootingare determined. The portrait scene discrimination level is calculatedbased on the aforementioned equation.

In the case where the portrait discrimination level is greater than orequal to a predetermined threshold value, the aperture value is set onthe open side, the exposure setting is set to +3 such that the color ofthe face is brightened, the contrast setting is set to −3 such thatgradation is smooth, and the finishing setting is set to portrait,according to the portrait discrimination level.

In step S502, the landscape scene discrimination level is calculated,and, in the case where the landscape scene discrimination level isgreater than or equal to a predetermined threshold value, the landscapeshooting settings are determined. In landscape scene discrimination, itis determined to be a landscape scene in the case where the brightnessdetermination result is bright, the object distance is greater than orequal to a predetermined distance, the sum of the color blocks of FIG.14 other than white, black, gray and skin color is greater than or equalto a predetermined rate, such as occupying 30%, for example. In the casewhere it is determined to be a landscape scene, the shooting settingsare set to landscape, saturation and contrast are set to +3, and theexposure value is set to −1.

In step S503, the evening view level scene discriminability iscalculated, and, in the case where the evening view level scenediscriminability is greater than or equal to a predetermined thresholdvalue, it is determined to be an evening view shooting setting. Inevening view scene discrimination, the determination result of theaforementioned evening view determination is used. In the case where itis determined to be an evening view, the exposure is set to −2, thesaturation setting is set to +2, and the coloration is set to shady.

In step S504, the night view level scene discriminability is calculated,and, in the case where the night view level scene discrimination isgreater than or equal to a predetermined threshold value, it isdetermined to be a night view shooting setting. In night view scenedetermination, the night view determination result is used. In the casewhere it is determined to be a night view, if a face is not detected,exposure is set to −2 and contrast is set to +2. If a face is detected,exposure is set to −1 and contrast is set to +1. In the case where aface is detected, the settings are configured to avoid overdoing shadowline enhancement (kuroshime), which tends to result in the face becomingwashed out, and in the case where a face is not detected, the settingsare configured to increase shadow line enhancement and create a strikingimage.

In step S505, high key shooting setting determination processing isperformed. In high key shooting determination, high key shooting is setin the case where brightness is determined to be high and the number ofthe color blocks that are determined to be white is greater than orequal to a predetermined percentage within the screen. Specifically, itis determined to be high key shooting in the case where the followingconditions are satisfied:

Condition 1: white blocks≥10%

Condition 2: determined to be bright

Condition 3: black blocks≤10%, white blocks≤60%

The high key discrimination level is determined using the percentage ofwhite block. The high key level is determined to be 30% at 10% whiteblocks, and to be 70% at 30% white blocks. In the case where it isdetermined to be high key, the exposure setting is set to +6, saturationis set to −3, and contrast is set to −2.

In step S506, low key shooting setting determination processing isperformed. In low key shooting determination, low key shooting is set inthe case where the shooting scene has a contrast greater than or equalto a predetermined value and a halftone gradation, and the number ofblack blocks and white blocks is greater than or equal to apredetermined value. Specifically, it is determined to be low keyshooting in the case where the following conditions are satisfied.

Condition 1: 10%≤white block≤30%

Condition 2: 10%≤black block≤30%

Condition 3: contrast (luminance standard deviation of image)≥50

The low key discrimination level is set to 50% in the case of satisfyingthe above determination. In the case where it is not determined to below key, the exposure setting is set to −6, saturation is set to +3, andcontrast is set to +2.

In step S507, monochrome shooting setting determination processing isperformed. In monochrome shooting determination, monochrome shooting isset in the case where the shooting scene has a contrast greater than orequal to a predetermined value and the number of halftone gradationblocks and white blocks is greater than or equal to a predeterminedvalue. Specifically, it is determined to be monochrome shooting in thecase where the following conditions are satisfied.

Condition 1: 10%≤white block≤30%

Condition 2: Blocks other than white blocks and black blocks occupy 50%of screen.

Condition 3: Contrast (luminance standard deviation of image)≥50. Themonochrome discrimination level is set to 40%, in the case of satisfyingthe above determination.

In the case where it is determined to be monochrome, and a face has notbeen detected or the size of the face is less than or equal to apredetermined size, it is determined to configure the following settingsbased on the color block rate. The number of red blocks, the number ofblue blocks, and the total number of yellow blocks and skin color blocksare calculated, and pink is set if the red blocks are the most numerous,purple is set if blue blocks are numerous, sepia is set if theyellow/skin color blocks are numerous, and black and white is set in allother cases. In the case where the size of the face is greater than orequal to a given size, black and white is set, and contrast and exposurecorrection are respectively set to 4 and −2.

Next, in step S303 of FIG. 16A, color effect proposition extraction isperformed. Color effect proposition involves determining color effectsettings and shooting settings suitable for a scene, based on analyzedscene information.

FIG. 16D is a flowchart showing the flow of operations for determiningthe contents of color effect proposition. Steps S601 to S604 show filterdetermination processing of the respective color, and description willbe given using a representative example, since the processing contentsfor each color are the same, and only the parameters differ.

FIGS. 17A and 17B are diagrams showing color detection settings, andscene discrimination levels and shooting settings in the case where therespective colors are detected. In the present embodiment, settings foryellow and blue will be described, and since the other colors can besimilarly set, description thereof is omitted in the present embodiment.

In step S601 of FIG. 16D, shooting setting determination processing foryellow color effect settings is performed. In determination of yellowcolor effect settings, as shown in FIG. 17A, in the case where Bv is 6or more and the percentage of yellow blocks occupying the screen is 40%or more, it is determined to be a yellow color effect fitting scene, andthe scene discrimination level is determined at 50%. In terms ofsettings, exposure is set to +3, the coloration setting is set to shady,and coloration correction (A−B) is set to +7. Since yellow objects oftenhave high reflectance and exposure is often controlled to underexpose,the yellowish tinge over the entire screen is enhanced by settingexposure on the high side and setting WB to a high color temperature.When shooting settings are configured in this way, the overall colortends to be on the amber side, and a warm twilight image can berepresented.

Similarly, in the case where Bv is 6 or more and yellow blocks occupy20% or more, the scene discrimination level is determined at 40%. Interms of settings, exposure is set to +3, contrast is set to +3,saturation is set to +2, the coloration setting is set to sunny, andcoloration correction (A−B) is set to +9. By configuring the settings inthis way, an image imbued with amber and wrapped in morning light can beshot.

In the case where Bv is 6 or more, yellow blocks occupy 10% or more andless than 20%, and it is determined to be macro, the scenediscrimination level is determined as 30%. With the settings in thiscase, exposure is set to −2, saturation is set to −2, the colorationsetting is set to shady, coloration correction (A−B) is set to +7,coloration correction (G−Mg) is set to +4. By configuring the settingsin this way, an image with a slightly dark and retro feel while exudingwarmth with amber can be shot.

Similarly, shooting setting determination processing for green coloreffect is performed in step S602, shooting setting determinationprocessing for blue color effect setting is performed in step S603, andshooting setting determination processing for magenta color effect isperformed in step S604.

Note that FIG. 18 is a diagram showing the relationship betweenclassification of discriminated scenes and proposition type. As hasalready been described above, a shooting effect is proposed byperforming processing such as shown in FIG. 18 on the image, accordingto the scene classification, and showing the effect thereof to the user.

Next, in step S304 of FIG. 16A, shooting proposition determinationprocessing is performed. FIG. 16E is a flowchart showing detailedoperations of the shooting proposition determination processing of stepS304.

In FIG. 16E, it is checked in step S701 whether there is a pairproposition. If there is a pair proposition, the pair proposition ispreferentially employed. If a pair proposition does not exist, theprocessing proceeds to step S702, and the scene proposition contentsdetermined to fit the scene as a result of the scene determination,among the single propositions, are employed. In the case where there arethree or more scene fitting results, the highest two types of scenediscrimination levels are employed. In step S703, a color effect fittingscene is extracted. The remaining proposition contents other than thepair proposition of step S701 and the propositions of step S702 areemployed as proposition contents of a number selectable from those indescending order of scene fitness.

In the case where the required number of the proposition contents is notobtained in the above proposition processing, the required number ofpropositions are selected at random from among high key, low key,monochrome, and color effect propositions. Also, the proposition effectsthat are employed in the above propositions are not subject to randomselection. In the case where monochrome/color effect propositions havealready been similarly proposed, the color effect propositions formonochrome and similar colors is excluded from candidates of randomextraction.

Furthermore, when three shooting settings are determined, the degree ofsimilarity of the candidates is calculated. A difference value iscalculated by acquiring the difference for each shooting setting andmultiplying the acquired difference by a weight for each item. As theweights of the items, the weights of the brightness setting, thebackground bokeh setting, the finishing setting and the colorationsetting, which are settings that greatly affect the image, areincreased, and the weights of object blur, coloration correction (A−B),coloration correction (G−Mg), crispness and vividness are reduced. Thedifferences of these shooting settings are multiplied by weights, and inthe case where the resultant value is not greater than or equal to apredetermined difference, low priority shooting setting candidates aredeleted and new shooting settings are extracted in the aforementionedshooting setting extraction flow.

As aforementioned, shooting settings are determined by scene fitness,and image candidates are displayed according to scene fitness.Specifically, in the example of the screen in FIG. 6B, the candidatesare arranged in setting candidates 1, 2, and 3 in descending order ofscene fitness. Also, in the case where a pair proposition is made,images of the pair proposition are arranged in candidate 1 and candidate2. In the case where setting of a single proposition is selected, singlepropositions and color effect propositions having high scene fitness arearranged. Randomly selected candidates are arranged as low ordercandidates. By performing arrangement in this way, the user can be moreeffectively made aware of the shooting candidates.

Note that, although not described in the present embodiment, a shutterspeed may be proposed after performing motion determination. In the casewhere it is determined that panning is being performed or in the casewhere a region that is constantly carrying out the same motion, anoptimal shutter speed that does not causes object blur to occur iscalculated, from the motion amount between frames and the frameinterval, and pair proposition of the optimal shutter speed and a slowershutter speed is performed. Also, in the case where motion of an objectis detected, shutter speed proposition that does not cause object blurto occur may be performed.

Also, in the case where the depth difference between the main object andthe background object is small and it is determined that the amount ofbackground blur is small, bokeh can be increased by dividing the mainobject region and the background region, defocusing the backgroundregion by image processing based on distance information, andcompositing the processed background region with the main object region.Similarly, in the case where it is determined that the main objectexists near the center of the screen, toy camera-like processing thatgreatly reduces the peripheral brightness may be adapted. Also, in thecase where it is determined to be landscape shooting, processing such asapplying a diorama-like filter that performs defocus processing thatleaves a portion of the lower screen and strongly defocuses regionsseparated from that portion.

Note that it is also possible to perform similar processing to thepresent embodiment at the time of editing a RAW image or a YUV imageobtained in the actual shooting. The image obtained in the actualshooting is read out from memory and undergoes scene analysis, and aplurality of settings that depend on the analyzed scene are selected. Itis sufficient to generate a plurality of images to which processing thatdepends on each of the plurality of selected settings has been applied,and to display these images together with the images read out frommemory. Since this is, however, not processing at the time of shooting,shutter speed or aperture cannot be set. In view of this, it issufficient to apply digital gain and background defocus processing,instead of setting shutter speed and aperture. Also, it is sufficient toembed the object distance in metadata together with the image at thetime of actual shooting.

OTHER EMBODIMENTS

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2016-129132, filed Jun. 29, 2016, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image capturing apparatus comprising: at leastone non-transitory memory device; at least one processor; a cameradevice that includes a taking lens and an image sensor and captures afirst image, based on a first shooting setting; a scene discriminationunit that discriminates a scene of an object, based on the firstshooting setting and a feature amount of the first image; a candidategeneration unit that generates a shooting setting candidate, based on aresult of the discrimination by the scene discrimination unit; an imagegeneration unit that generates, based on the shooting setting candidategenerated by the candidate generation unit, an image reflecting aneffect obtained by the shooting setting candidate; and a display controlunit that causes a display device to display the image generated by theimage generation unit, wherein the candidate generation unit performs atleast one of a first operation for generating a plurality of shootingsetting candidates in which a predetermined parameter is separated bygreater than or equal to a threshold value out of the shooting settingsbased on a result of the discrimination by the scene discrimination unitand a second operation for generating at least one shooting settingcandidate based on the result of the discrimination by the scenediscrimination unit, and the display control unit causes the displaydevice to preferentially display the shooting setting candidatesgenerated by the first operation, and wherein the scene discriminationunit, the candidate generation unit, the image generation unit and thedisplay control unit are implemented by the at least one processorexecuting at least one program recorded on the at least onenon-transitory memory device.
 2. The image capturing apparatus accordingto claim 1, further comprising a selection member for a user to selectone image from the plurality of images displayed by the display device.3. The image capturing apparatus according to claim 2, furthercomprising a control unit that causes the camera device to performshooting using shooting settings corresponding to the image selected bythe selection member, wherein the control unit is implemented by the atleast one processor executing at least one program recorded on the atleast one non-transitory memory device.
 4. The image capturing apparatusaccording to claim 1, wherein the image generation unit generates imagesreflecting effects obtained by the shooting setting candidates, byperforming image processing on the first image.
 5. The image capturingapparatus according to claim 1, wherein the candidate generation unitgenerates two shooting setting candidates by the first operation and atleast one shooting setting candidate by the second operation, and thedisplay control unit causes the display device to display the pluralityof shooting setting candidates generated by the first operation and thesecond operation, in a case where the shooting setting candidates aregenerated by the first operation, and causes the display device todisplay the shooting setting candidate generated by the secondoperation, in a case where the shooting setting candidates are notgenerated by the first operation.
 6. The image capturing apparatusaccording to claim 5, further comprising a determination unit thatdetermines a degree of similarity with regard to the shooting settingcandidates generated with the first operation and the shooting settingcandidate generated with the second operation, wherein the displaycontrol unit causes the display device to display the shooting settingcandidates that are not determined to have a high degree of similarityby the determination unit, and wherein the determination unit isimplemented by the at least one processor executing at least one programrecorded on the at least one non-transitory memory device.
 7. The imagecapturing apparatus according to claim 1, wherein the candidategeneration unit generates shooting setting candidates having differentaperture values or exposures, in a case where any of portrait shooting,backlight shooting and macro shooting is discriminated by the scenediscrimination unit.
 8. The image capturing apparatus according to claim7, wherein, in the first operation, the candidate generation unitgenerates a shooting setting candidate with which an aperture is openand a shooting setting candidate with which the aperture is closed bymore than a predetermined value, in the case where portrait shooting isdiscriminated by the scene discrimination unit.
 9. The image capturingapparatus according to claim 7, wherein the candidate generation unitgenerates a shooting setting candidate with which a main object iscorrectly exposed and a shooting setting candidate with which abackground is correctly exposed, in the case where backlight shooting isdiscriminated by the scene discrimination unit.
 10. The image capturingapparatus according to claim 7, wherein the candidate generation unitgenerates a shooting setting candidate with which an aperture is openand a shooting setting candidate with which the aperture is closed bymore than a predetermined value, in the case where macro shooting isdiscriminated by the scene discrimination unit.
 11. The image capturingapparatus according to claim 1, wherein the candidate generation unitgenerates the shooting setting candidate by the second operation, in thecase where any of landscape shooting, evening view shooting, night viewshooting, high key shooting, low key shooting and monochrome shooting isdiscriminated by the scene discrimination unit.
 12. The image capturingapparatus according to claim 1, wherein the candidate generation unitgenerates the shooting setting candidate based on an in-plane colordistribution, in the second operation.
 13. The image capturing apparatusaccording to claim 12, wherein the candidate generation unit generatesthe shooting setting candidate based on green, yellow, blue and magentacolor distributions, in the second operation.
 14. An image processingapparatus comprising: at least one non-transitory memory device; atleast one processor; a scene discrimination unit that discriminates ascene of an object, based on a feature amount of a first image; acandidate generation unit that generates a setting candidate relating toat least one of a shooting parameter and an image processing parameter,based on a result of the discrimination by the scene discriminationunit; an image generation unit that generates, based on the settingcandidate generated by the candidate generation unit, an imagereflecting an effect obtained by the setting candidate; and a displaycontrol unit that causes a display device to display the image generatedby the image generation unit, wherein the candidate generation unitperforms at least one of a first operation for generating a plurality ofsetting candidates in which a predetermined parameter is separated bygreater than or equal to a threshold value out of the settings based ona result of the discrimination by the scene discrimination unit and asecond operation for generating at least one setting candidate based onthe result of the discrimination by the scene discrimination unit, andthe display control unit causes the display device to preferentiallydisplay the setting candidates generated by the first operation, andwherein the scene discrimination unit, the candidate generation unit,the image generation unit and the display control unit are implementedby the at least one processor executing at least one program recorded onthe at least one non-transitory.
 15. The image processing apparatusaccording to claim 14, further comprising a selection member for a userto select one image from the plurality of images displayed by thedisplay device.
 16. The image processing apparatus according to claim15, further comprising a control unit that causes the camera device toperform shooting using settings corresponding to the image selected bythe selection member, wherein the control unit is implemented by the atleast one processor executing at least one program recorded on the atleast one non-transitory memory device.
 17. The image processingapparatus according to claim 14, wherein the image generation unitgenerates images reflecting effects obtained by the setting candidates,by performing image processing on the first image.
 18. A method ofcontrolling an image capturing apparatus that includes an imagecapturing unit that captures a first image based on a first shootingsetting, the method comprising: discriminating a scene of an object,based on the first shooting setting and a feature amount of the firstimage; generating a shooting setting candidate, based on a result of thediscrimination in the scene discrimination; generating, based on theshooting setting candidate generated in the candidate generation, animage reflecting an effect obtained by the shooting setting candidate;and displaying the image generated in the image generation, wherein, inthe candidate generation, at least one of a first operation forgenerating a plurality of shooting setting candidates in which apredetermined parameter is separated by greater than or equal to athreshold value out of the shooting settings based on a result of thediscrimination in the scene discrimination and a second operation forgenerating at least one shooting setting candidate based on the resultof the discrimination in the scene discrimination is performed, and inthe display, the shooting setting candidates generated by the firstoperation are preferentially displayed.
 19. A control method for animage processing apparatus, the method comprising: discriminating ascene of an object, based on a feature amount of a first image;generating a setting candidate relating to at least one of a shootingparameter and an image processing parameter, based on a result of thediscrimination in the scene discrimination; generating, based on thesetting candidate generated in the candidate generation, an imagereflecting an effect obtained by the setting candidate; and displayingthe image generated in the image generation, wherein, in the candidategeneration, at least one of a first operation for generating a pluralityof setting candidates in which a predetermined parameter is separated bygreater than or equal to a threshold value out of the settings based ona result of the discrimination in the scene discrimination and a secondoperation for generating at least one setting candidate based on theresult of the discrimination in the scene discrimination is performed,and in the display, the setting candidates generated by the firstoperation is preferentially displayed.